Feat/background goroutine get job info test #4368

fscnick · 2026-01-12T15:40:50Z

Why are these changes needed?

This PR provides the unit test and e2e test for #4160.

it uses synctest for the unit test
it applies the feature gate to run the existed e2e test.

Related issue number

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

… for cache Signed-off-by: fscnick <[email protected]>

Signed-off-by: fscnick <[email protected]>

fscnick · 2026-01-12T15:44:22Z

ray-operator/controllers/ray/utils/dashboardclient/dashboard_cache_client_test.go

+		// Test with getting a persistent error, the cache should be removed eventually.
+		nonExistedJobId := "not-existed-job-id"


Another test in the same bubble. The goroutine is created by singleton and synctest is not allowed to access goroutine outside of bubble.

fscnick · 2026-01-12T15:45:28Z

ray-operator/test/e2erayjob/rayjob_deletion_strategy_test.go

 				WithRayClusterSpec(NewRayClusterSpec(MountConfigMap[rayv1ac.RayClusterSpecApplyConfiguration](jobs, "/home/ray/jobs"))).
 				WithEntrypoint("python /home/ray/jobs/long_running.py").
-				WithActiveDeadlineSeconds(45).       // Short deadline for failing the JobDeploymentStatus, but making sure the cluster is running
+				WithActiveDeadlineSeconds(60).       // Short deadline for failing the JobDeploymentStatus, but making sure the cluster is running


Add a few second to avoid flaky.

fscnick · 2026-01-12T15:48:36Z

ray-operator/test/e2erayjob/rayjob_deletion_strategy_test.go

+		g.Eventually(RayJob(test, rayJob.Namespace, rayJob.Name), TestTimeoutMedium).
+			Should(And(
+				WithTransform(RayJobStatus, Equal(rayv1.JobStatusRunning)),
+				WithTransform(RayJobReason, Equal(rayv1.DeadlineExceeded))))


use Eventually to wait for the condition satisfied.

…oroutine-get-job-info-test

ray-operator/controllers/ray/utils/dashboardclient/dashboard_cache_client_test.go

Signed-off-by: fscnick <[email protected]>

Future-Outlier · 2026-01-14T06:30:23Z

cursor review

cursor

✅ Bugbot reviewed your changes and found no bugs!

Future-Outlier

cc @seanlaii @400Ping to take a look

Future-Outlier · 2026-01-14T08:29:51Z

cc @AndySung320 to take a look

Future-Outlier

Should we only keep testing AsyncJobInfoQuery=false?
since this is not yet in beta version or GA?

Future-Outlier

lgtm, cc @rueian to merge
but I personally think we should keep testing AsyncJobInfoQuery=false

#4368 (review)

…oroutine-get-job-info-test

fscnick · 2026-01-14T14:43:14Z

Should we only keep testing AsyncJobInfoQuery=false? since this is not yet in beta version or GA?

I'm not that certain about it, kindly let me know if I was wrong.

Isn't AsyncJobInfoQuery=false for the nightly build ? Add it to the test for ensuring the new feature doesn't break the existed test.

kuberay/.buildkite/build-start-operator.sh

Lines 15 to 20 in a9a4ab0

    
           else 
        
               IMG=kuberay/operator:nightly make docker-image && 
        
               kind load docker-image kuberay/operator:nightly && 
        
               echo "Deploying operator with test overrides (feature gates via test-overrides overlay)" 
        
               IMG=kuberay/operator:nightly make deploy-with-override 
        
           fi

rueian · 2026-01-14T15:25:28Z

Isn't AsyncJobInfoQuery=false for the nightly build ? Add it to the test for ensuring the new feature doesn't break the existed test.

Can we test both AsyncJobInfoQuery=false and AsyncJobInfoQuery=true?

…oQuery Signed-off-by: fscnick <[email protected]>

fscnick · 2026-01-15T15:24:36Z

.buildkite/test-e2e.yml

+    # Build nightly KubeRay operator image
+    - pushd ray-operator
+    - IMG=kuberay/operator:nightly make docker-image && kind load docker-image kuberay/operator:nightly && echo "Deploying operator with test overrides without AsyncJobInfoQuery (feature gates via test-overrides-without-async-job-info-query overlay )"
+    - IMG=kuberay/operator:nightly make deploy-with-override-without-async-job-info-query


Add an additional e2e test with a different make command to test them with AsyncJobInfoQuery=true

ray-operator/config/overlays/test-overrides-with-async-job-info-query/deployment-override.yaml

Signed-off-by: fscnick <[email protected]>

…oroutine-get-job-info-test

fscnick added 17 commits January 12, 2026 14:39

[RayJob] background job info poc

1bae584

[RayJob] encapsulate the worker pool

293937b

[RayJob] replace concurrency map with lru cache

b4326bc

[RayJob] remove cache on stop and config flag

fef0c77

[RayJob] remove delete cache from deleteClusterResources and add lock…

13070f0

… for cache Signed-off-by: fscnick <[email protected]>

[Helm] add argument for useBackgroundGoroutine

3d07403

Signed-off-by: fscnick <[email protected]>

[RayJob] remove unused function and background goroutine observability

5db324b

Signed-off-by: fscnick <[email protected]>

[RayJob] rename useBackgroundGoroutine to asyncJobInfoQuery

026e9f0

Signed-off-by: fscnick <[email protected]>

[RayJob] make cache immutable to avoid data race

645aaed

Signed-off-by: fscnick <[email protected]>

[RayJob] remove unused function

bcb2a38

Signed-off-by: fscnick <[email protected]>

[RayJob] If error on fetching job info, it removes from loop

6dc8cf6

Signed-off-by: fscnick <[email protected]>

[RayJob] task queue is extendable

b0b2753

Signed-off-by: fscnick <[email protected]>

[RayJob] change slice to ring buffer

db5aa09

Signed-off-by: fscnick <[email protected]>

[RayJob] async job info query use feature gate instead

98a17d1

Signed-off-by: fscnick <[email protected]>

[RayJob] add test for async job info query

38a8602

Signed-off-by: fscnick <[email protected]>

[Test][RayJob] fix e2e test and add unit test

adc8003

Signed-off-by: fscnick <[email protected]>

[RayJob] remove redundent code in async job query

31eea56

Signed-off-by: fscnick <[email protected]>

fscnick commented Jan 12, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/master' into feat/background-g…

8231132

…oroutine-get-job-info-test

fscnick marked this pull request as ready for review January 13, 2026 11:50

fscnick requested review from MortalHappiness, andrewsykim, kevin85421 and rueian as code owners January 13, 2026 11:50

cursor bot reviewed Jan 13, 2026

View reviewed changes

ray-operator/controllers/ray/utils/dashboardclient/dashboard_cache_client_test.go Show resolved Hide resolved

[Test][RayJob] fix cache key name in test

82de99a

Signed-off-by: fscnick <[email protected]>

fscnick force-pushed the feat/background-goroutine-get-job-info-test branch 2 times, most recently from f8c8afb to 5ea0272 Compare January 14, 2026 00:45

cursor bot reviewed Jan 14, 2026

View reviewed changes

Future-Outlier reviewed Jan 14, 2026

View reviewed changes

Future-Outlier approved these changes Jan 14, 2026

View reviewed changes

Future-Outlier moved this to can be merged in @Future-Outlier's kuberay project Jan 14, 2026

Future-Outlier added this to @Future-Outlier's kuberay project Jan 14, 2026

Merge remote-tracking branch 'upstream/master' into feat/background-g…

18d6c84

…oroutine-get-job-info-test

fscnick force-pushed the feat/background-goroutine-get-job-info-test branch from 5ea0272 to 18d6c84 Compare January 14, 2026 14:22

[Test] e2e RayJob test with AsyncJobInfoQuery and without AsyncJobInf…

b64d60c

…oQuery Signed-off-by: fscnick <[email protected]>

fscnick commented Jan 15, 2026

View reviewed changes

cursor bot reviewed Jan 15, 2026

View reviewed changes

ray-operator/config/overlays/test-overrides-with-async-job-info-query/deployment-override.yaml Show resolved Hide resolved

fscnick added 2 commits January 15, 2026 23:33

[Test] rename to with AsyncJobInfoQuery for testing

f2f717f

Signed-off-by: fscnick <[email protected]>

Merge remote-tracking branch 'upstream/master' into feat/background-g…

720dee5

…oroutine-get-job-info-test

		// Test with getting a persistent error, the cache should be removed eventually.
		nonExistedJobId := "not-existed-job-id"

Feat/background goroutine get job info test #4368

Are you sure you want to change the base?

Feat/background goroutine get job info test #4368

Conversation

fscnick commented Jan 12, 2026

Why are these changes needed?

Related issue number

Checks

Uh oh!

fscnick Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

fscnick Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

fscnick Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Future-Outlier commented Jan 14, 2026

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

Future-Outlier commented Jan 14, 2026

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

Future-Outlier left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fscnick commented Jan 14, 2026

Uh oh!

rueian commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fscnick Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Future-Outlier left a comment •

edited

Loading

rueian commented Jan 14, 2026 •

edited

Loading